HarvestCycle encounters a failure when `NewRelic.manual_shutdown()` is invoked after test case completion.

[Harish-M]

Describe the bug
The following logs are produced when the test cases are executed, and after consolidating the results, additional logs are generated.

After removing the call to NewRelic.manual_shutdown(), the error logs no longer persist.

Additionally, these error logs are not hindering any actions.

Note:
These error logs did not appear in version 1.30.0. However, after upgrading the Elixir agent to version 1.38.0, the error logs started to persist.

Environment

• Elixir & Erlang version (elixir -v): 1.15.7-otp-26
• Agent version (mix deps | grep new_relic_agent):

`12:44:46 07:14:46.252 [error] Task #PID<0.60984.0> started from #PID<0.60982.0> terminating
12:44:46 ** (stop) exited in: GenServer.call(NewRelic.Harvest.Collector.TransactionTrace.HarvestCycle, :pause, 5000)
12:44:46 ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
12:44:46 (elixir 1.15.7) lib/gen_server.ex:1063: GenServer.call/3
12:44:46 (new_relic_agent 1.34.0) lib/new_relic/harvest/harvest_cycle.ex:51: NewRelic.Harvest.HarvestCycle.manual_shutdown/1
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
12:44:46 Function: #Function<1.83909589/0 in NewRelic.Harvest.Supervisor.manual_shutdown/0>
12:44:46 Args: []

12:44:46 ** (EXIT from #PID<0.60982.0>) exited in: GenServer.call(NewRelic.Harvest.Collector.TransactionTrace.HarvestCycle, :pause, 5000)
12:44:46 ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
12:44:46
12:44:46
12:44:46 07:14:46.252 [error] Task #PID<0.60985.0> started from #PID<0.60982.0> terminating
12:44:46 ** (stop) exited in: GenServer.call(NewRelic.Harvest.Collector.TransactionEvent.HarvestCycle, :pause, 5000)
12:44:46 ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
12:44:46 (elixir 1.15.7) lib/gen_server.ex:1063: GenServer.call/3
12:44:46 (new_relic_agent 1.34.0) lib/new_relic/harvest/harvest_cycle.ex:51: NewRelic.Harvest.HarvestCycle.manual_shutdown/1
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
12:44:46 Function: #Function<1.83909589/0 in NewRelic.Harvest.Supervisor.manual_shutdown/0>
12:44:46 Args: []
12:44:46
12:44:46 07:14:46.252 [error] Task #PID<0.60988.0> started from #PID<0.60982.0> terminating
12:44:46 ** (stop) exited in: GenServer.call(NewRelic.Harvest.Collector.CustomEvent.HarvestCycle, :pause, 5000)
12:44:46 ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
12:44:46 (elixir 1.15.7) lib/gen_server.ex:1063: GenServer.call/3
12:44:46 (new_relic_agent 1.34.0) lib/new_relic/harvest/harvest_cycle.ex:51: NewRelic.Harvest.HarvestCycle.manual_shutdown/1
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
12:44:46 Function: #Function<1.83909589/0 in NewRelic.Harvest.Supervisor.manual_shutdown/0>
12:44:46 Args: []
12:44:46
12:44:46 07:14:46.252 [error] Task #PID<0.60987.0> started from #PID<0.60982.0> terminating
12:44:46 ** (stop) exited in: GenServer.call(NewRelic.Harvest.Collector.TransactionErrorEvent.HarvestCycle, :pause, 5000)
12:44:46 ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
12:44:46 (elixir 1.15.7) lib/gen_server.ex:1063: GenServer.call/3
12:44:46 (new_relic_agent 1.34.0) lib/new_relic/harvest/harvest_cycle.ex:51: NewRelic.Harvest.HarvestCycle.manual_shutdown/1
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
12:44:46 Function: #Function<1.83909589/0 in NewRelic.Harvest.Supervisor.manual_shutdown/0>
12:44:46 Args: []
12:44:46
12:44:46 07:14:46.253 [error] Task #PID<0.60989.0> started from #PID<0.60982.0> terminating
12:44:46 ** (stop) exited in: GenServer.call(NewRelic.Harvest.Collector.ErrorTrace.HarvestCycle, :pause, 5000)
12:44:46 ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
12:44:46 (elixir 1.15.7) lib/gen_server.ex:1063: GenServer.call/3
12:44:46 (new_relic_agent 1.34.0) lib/new_relic/harvest/harvest_cycle.ex:51: NewRelic.Harvest.HarvestCycle.manual_shutdown/1
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
12:44:46 (elixir 1.15.7) lib/task/supervised.ex:36: Task.Supervised.reply/4
12:44:46 Function: #Function<1.83909589/0 in NewRelic.Harvest.Supervisor.manual_shutdown/0>
12:44:46 Args: []`

[Harish-M]

@binaryseed, The issue has not been addressed in the mentioned change. I believe switching to the proposed method might resolve the problem. Could you please verify if the proposed solution fixes the reported issue?

  def manual_shutdown(harvest_cycle) do
    # Check if the harvest cycle process is still alive before attempting to pause it
    if Process.alive?(harvest_cycle) do
      case current_harvester(harvest_cycle) do
        nil ->
          :ignore

        harvester ->
          Process.monitor(harvester)
          
          # Use a try-catch to handle race conditions where process dies between alive check and call
          try do
            GenServer.call(harvest_cycle, :pause)

            receive do
              {:DOWN, _ref, _, ^harvester, _reason} ->
                NewRelic.log(:warning, "Completed shutdown #{inspect(harvest_cycle)}")
            end
          catch
            :exit, {:noproc, _} ->
              NewRelic.log(:debug, "Harvest cycle #{inspect(harvest_cycle)} already shutdown during manual_shutdown")
            :exit, {:normal, _} ->
              NewRelic.log(:debug, "Harvest cycle #{inspect(harvest_cycle)} shutdown normally during manual_shutdown")
          end
      end
    else
      NewRelic.log(:debug, "Harvest cycle #{inspect(harvest_cycle)} already dead, skipping manual shutdown")
      :ignore
    end
  end