MONO Server Freezes After 2 Days

[marek-safar]

From @mysteryx93 on February 11, 2018 0:10

I had been having issues with MONO taking 100% CPU on both Ubuntu and Debian with Apache2 and mod_mono. As a last resort, I switched Apache2 to worker mode and started the MONO servers applications manually. It's working smoothly for a day or two, then the MONO servers stop responding. This might be the same issue as before, but reacting differently -- freezing instead of taking 100% CPU.

However, this time I have detailed log.

First this entry appears 3 times

(info) Auto generated encryption keys not saved: System.Security.SecurityException: No access to the given key ---> System.UnauthorizedAccessException: Access to the path "/var/www/.mono/registry" is denied.
  at System.IO.Directory.CreateDirectoriesInternal (System.String path) [0x0005e] in <2e7c1c96edae44d496118948ca617c11>:0 
  at System.IO.Directory.CreateDirectory (System.String path) [0x0008f] in <2e7c1c96edae44d496118948ca617c11>:0 
  at System.IO.DirectoryInfo.Create () [0x00000] in <2e7c1c96edae44d496118948ca617c11>:0 
  at (wrapper remoting-invoke-with-check) System.IO.DirectoryInfo.Create()
  at System.IO.Directory.CreateDirectoriesInternal (System.String path) [0x00036] in <2e7c1c96edae44d496118948ca617c11>:0 
  at System.IO.Directory.CreateDirectory (System.String path) [0x0008f] in <2e7c1c96edae44d496118948ca617c11>:0 
  at Microsoft.Win32.KeyHandler..ctor (Microsoft.Win32.RegistryKey rkey, System.String basedir, System.Boolean is_volatile) [0x00038] in <2e7c1c96edae44d496118948ca617c11>:0 
   --- End of inner exception stack trace ---
  at Microsoft.Win32.KeyHandler..ctor (Microsoft.Win32.RegistryKey rkey, System.String basedir, System.Boolean is_volatile) [0x0004d] in <2e7c1c96edae44d496118948ca617c11>:0 
  at Microsoft.Win32.KeyHandler..ctor (Microsoft.Win32.RegistryKey rkey, System.String basedir) [0x00000] in <2e7c1c96edae44d496118948ca617c11>:0 
  at Microsoft.Win32.KeyHandler.Lookup (Microsoft.Win32.RegistryKey rkey, System.Boolean createNonExisting) [0x0008e] in <2e7c1c96edae44d496118948ca617c11>:0 
  at Microsoft.Win32.UnixRegistryApi.OpenSubKey (Microsoft.Win32.RegistryKey rkey, System.String keyname, System.Boolean writable) [0x00000] in <2e7c1c96edae44d496118948ca617c11>:0 
  at Microsoft.Win32.RegistryKey.OpenSubKey (System.String name, System.Boolean writable) [0x0001b] in <2e7c1c96edae44d496118948ca617c11>:0 
  at (wrapper remoting-invoke-with-check) Microsoft.Win32.RegistryKey.OpenSubKey(string,bool)
  at System.Web.Configuration.MachineKeyRegistryStorage.OpenRegistryKey (System.String path, System.Boolean write) [0x00021] in <52b8d38f1da445f7ad770be558c9fed4>:0 
  at System.Web.Configuration.MachineKeyRegistryStorage.Store (System.Byte[] buf, System.Web.Configuration.MachineKeyRegistryStorage+KeyType kt) [0x0002f] in <52b8d38f1da445f7ad770be558c9fed4>:0 

(Just fixed permissions on that folder)

Then the log says this tons of times

Failed to receive record.
Terminating connection.

And then this once

WARNING: WebConfigurationManager's LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)

The rest of the log is pages and pages and pages of

Failed to receive record.
Terminating connection.

Any idea what's going on here and how to fix it?

Copied from original issue: mono/mono#6941

[marek-safar]

From @mysteryx93 on February 12, 2018 20:11

After fixing permission issues, the first error is gone but the rest is still the same, it froze and after restarting the log contains this

Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
WARNING: WebConfigurationManager's LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.
Failed to receive record.
Terminating connection.

[marek-safar]

From @mysteryx93 on February 14, 2018 2:55

If it matters, this is the command used to start the server

sudo -H -u www-data /usr/bin/fastcgi-mono-server4 /applications=/:/var/www/spiritualselftransformation/ /socket=tcp:127.0.0.1:9001 > /var/log/mono/spiritualselftransformation.log 2>&1 &

[mysteryx93]

In this case I'm not using mod_mono.

I however noticed that of 5 websites, 4 were freezing and 1 kept running. The one that kept running was configured to use .NET v4.7.1, while the 4 others were at v4.0

I updated them all to use v4.7.1, and made sure debug was disabled, in Web.config, and so far, only 1 froze and 4 kept running. I'm doing more tests, I think there's something in my web.config it doesn't like.

However there's definitely an issue with mod_mono. I'm not sure whether it's the same issue reacting differently or a separate issue.

[tdymel]

Hey!
I am recently experiencing the same issues!
I am not using mod_mono neither. Im using nginx with the fastcgi-server.
It feels like that the application itself is not crashing. Looks more like the fastcgi server is terminating the connection.
It usually happens after a day for me though. Depending on the amount of traffic Im receiving, its going faster.

@mysteryx93 ,
I already answered you on Stackoverflow. So I think I came up with a solution, though Im just running it since a few hours but so far so good.
It turns out that these errors where caused by a socket exception. I was previously using the tcp sockets, so I changed them up to unix sockets (Seem to have better performance anyway according to the web).

I will report back, once I had it running for a few days!

cheers!

Edit:
Reporting back. It has now been almost 24 hours. normally my application would have been freezing at that time, but it seems to be fine. No errors so far.

Hope you can fix this issue by that aswell.

[algida]

@Geigerkind We have same issues, does your solution work, or did you find some other way to solve it?

[tdymel]

Yes. After changing the sockets, it worked flawlessy.

[AlexRixhardson]

I am experiencing a similar issue. However, I am already using unix sockets. I am using Apache and mono combination. When the problem start occurring, I see lots of incoming HTTPS connections stuck in the CLOSE_WAIT state. Does anyone know what could be the issue?