Skip to content

Adding fix for incorrect log level bug for versions 2.10.1 & 2.10.3#253

Merged
Vraj2725 merged 2 commits intoaws:mainfrom
Vraj2725:main
May 9, 2025
Merged

Adding fix for incorrect log level bug for versions 2.10.1 & 2.10.3#253
Vraj2725 merged 2 commits intoaws:mainfrom
Vraj2725:main

Conversation

@Vraj2725
Copy link
Copy Markdown
Contributor

@Vraj2725 Vraj2725 commented May 5, 2025

Issue #, if available:
When using airflow versions 2.9.2 and above customer are unable to see error logs when they set their loglevel to WARNING or above. The root cause is that in subprocess.py, the line self.process_logger.info(line.decode("utf-8")) overrides the log level coming from airflow stream to INFO in subprocess.py.

Description of changes:

  • Setting process logger level to DEBUG to emit all logs and setting root logger to customer's log level to respect customer's log level.
  • Refactoring entrypoint.py to resolve circular dependency issue and breaking down its logic into different files so all files have single responsibility such as:
    • user_requirements.py: Installing user requirements.
    • execute_command.py: Logic for executing all the subprocesses
    • setup_environment.py: Running the startup script and setting up environment variables

Testing

  • Added unit tests for all refactored code.
  • Ran log heavy dag on the refactored image code by using local docker.
  • Exported environment variable using startup script and verified that it set correctly.

This PR is for V 2.10.1 and 2.10.3. Fix for v2.9.2 has already reached prod.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@Vraj2725 Vraj2725 marked this pull request as ready for review May 6, 2025 06:03
@Vraj2725 Vraj2725 requested a review from sudiptoghosh01 May 7, 2025 09:57
@Vraj2725 Vraj2725 merged commit da4324d into aws:main May 9, 2025
1 check passed
@Vraj2725 Vraj2725 mentioned this pull request May 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants