Open
Description
Hello,
I am finding that socket.connect when destination IP/Port pair is not used may lead to surprising memory usage. the memory profile result looks like below. It is weir that sock.connect uses around 100MB memory. The @profile decorator is added to _reconnect function from fluent-logger.
Line # Mem usage Increment Occurrences Line Contents
=============================================================
196 415.797 MiB 255.617 MiB 20000 @profile
197 def _reconnect(self):
198 415.797 MiB 8.027 MiB 20000 if not self.socket:
199 415.797 MiB 0.000 MiB 20000 try:
200 415.797 MiB 0.082 MiB 20000 if self.host.startswith('unix://'):
201 sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
202 sock.settimeout(self.timeout)
203 sock.connect(self.host[len('unix://'):])
204 else:
205 415.797 MiB 27.230 MiB 20000 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
206 415.797 MiB 1.723 MiB 20000 sock.settimeout(self.timeout)
207 # This might be controversial and may need to be removed
208 415.797 MiB 0.000 MiB 20000 sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
209 #sock.connect((self.host, self.port))
210 415.797 MiB 117.297 MiB 20000 sock.connect(("localhost", 24244))
211 415.797 MiB 4.129 MiB 20000 except Exception as e:
212 415.797 MiB 0.000 MiB 20000 try:
213 415.797 MiB 0.148 MiB 20000 sock.close()
214 except Exception: # pragma: no cover
215 pass
216 415.797 MiB 1.543 MiB 20000 raise e
217 else:
218 self.socket = sockThe script I am using is
#!/usr/bin/env python3
import tracemalloc
import logging
import logging.config
import time
import eventlet
tracemalloc.start()
eventlet.monkey_patch()
logging.config.fileConfig("logging.conf")
loggers = []
counter = 2
for i in range(counter):
name = "name%s" % i
loggers.append(logging.getLogger(name))
pool = eventlet.GreenPool()
for j in range(10000):
for i in range(counter):
pool.spawn(loggers[i].warning, "hello world from sjt")
time.sleep(4)
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)
pool.waitall()
total = sum(stat.size for stat in top_stats)
print("total: ", total)And logging.conf looks like below
[formatter_context]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
[formatter_default]
datefmt = %Y-%m-%d %H:%M:%S
format = %(message)s
[formatter_fluentd]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
[formatters]
keys = context,default,fluentd
[handler_fluentd]
args = ("openstack.cinder.api", 'localhost', 24234)
kwargs = {'bufmax': 0}
class = fluent.handler.FluentHandler
formatter = fluentd
[handler_null]
args = ()
class = logging.NullHandler
formatter = default
[handler_stderr]
args = (sys.stderr,)
class = StreamHandler
formatter = context
[handler_stdout]
args = (sys.stdout,)
class = StreamHandler
formatter = context
[handlers]
keys = stdout,stderr,null,fluentd
[logger_root]
handlers = fluentd
level = DEBUG
[loggers]
keys = rootTo run the script,
python3 -m memory_profiler fluent_test.pyVersion:
eventlet: 0.33.0fluent-logger: 0.10.0python3: 3.8.10