Possible race condition in concurrent instantiation of local storage class

[rmax]

I'm writing a custom agent (to be integrated with legacy software) and I'm reusing prefect.environments.storage.Local.

When writing tests for this and running then concurrently, our CI reported this error:

18:46:24  tests/test_agent.py:27: 
18:46:24  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
18:46:24  <string>:4: in __init__
18:46:24      ???
18:46:24  /usr/local/lib/python3.6/site-packages/<redacted>/agent.py:84: in __post_init__
18:46:24      self._storage = LocalStorage()
18:46:24          self       = <redacted>
18:46:24  /usr/local/lib/python3.6/site-packages/prefect/environments/storage/local.py:59: in __init__
18:46:24      os.makedirs(abs_directory)
18:46:24          __class__  = <class 'prefect.environments.storage.local.Local'>
18:46:24          abs_directory = '/root/.prefect/flows'
18:46:24          directory  = '/root/.prefect/flows'
18:46:24          kwargs     = {}
18:46:24          path       = None
18:46:24          self       = <Storage: Local>
18:46:24          stored_as_script = False
18:46:24          validate   = True
18:46:24  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
18:46:24  
18:46:24  name = '/root/.prefect/flows', mode = 511, exist_ok = False
18:46:24  
18:46:24      def makedirs(name, mode=0o777, exist_ok=False):
18:46:24          """makedirs(name [, mode=0o777][, exist_ok=False])
18:46:24      
18:46:24          Super-mkdir; create a leaf directory and all intermediate ones.  Works like
18:46:24          mkdir, except that any intermediate path segment (not just the rightmost)
18:46:24          will be created if it does not exist. If the target directory already
18:46:24          exists, raise an OSError if exist_ok is False. Otherwise no exception is
18:46:24          raised.  This is recursive.
18:46:24      
18:46:24          """
18:46:24          head, tail = path.split(name)
18:46:24          if not tail:
18:46:24              head, tail = path.split(head)
18:46:24          if head and tail and not path.exists(head):
18:46:24              try:
18:46:24                  makedirs(head, mode, exist_ok)
18:46:24              except FileExistsError:
18:46:24                  # Defeats race condition when another thread created the path
18:46:24                  pass
18:46:24              cdir = curdir
18:46:24              if isinstance(tail, bytes):
18:46:24                  cdir = bytes(curdir, 'ASCII')
18:46:24              if tail == cdir:           # xxx/newdir/. exists if xxx/newdir exists
18:46:24                  return
18:46:24          try:
18:46:24  >           mkdir(name, mode)
18:46:24  E           FileExistsError: [Errno 17] File exists: '/root/.prefect/flows'
18:46:24  
18:46:24  cdir       = '.'
18:46:24  exist_ok   = False
18:46:24  head       = '/root/.prefect'
18:46:24  mode       = 511
18:46:24  name       = '/root/.prefect/flows'
18:46:24  tail       = 'flows'
18:46:24  
18:46:24  /usr/local/lib/python3.6/os.py:220: FileExistsError

It seems there is a race condition between os.path.exists and os.makedirs between multiple processes (using pytest-xdist):

prefect/src/prefect/environments/storage/local.py

Lines 58 to 59 in b115311

	if not os.path.exists(abs_directory):
	os.makedirs(abs_directory)

Thoughts?

Edit: Related PR that uses exist_ok flag #3679

[rmax]

Fix done in #3679.