Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2439

Remove LocalityManager and container location information from JobModel

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.6
    • None
    • None

    Description

      Goal: Make JobModel immutable by removing locality information (container to host mapping) from the JobModel.

      Motivation:
      1. Enables us to move JobModel to public APIs and expose it in JobContext.
      2. Enables us to cache and serve serialized JobModel from the AM servlet to reduce AM overhead (memory, open connections, num threads) during container startup, esp. for jobs with a large number of containers (See: https://github.com/apache/samza/pull/1241)
      3. Removes tech debt: models should be immutable, and should not update themselves.
      4. Removes tech debt: makes current container location a first class concept for container scheduling / placement , and for tools like dashboard, samza-rest, auto-scaling, diagnostics etc.

      Proposal:
      Move locality information (and LocalityManager) out of the JobModel. New JobModel should only contain Config and ContainerModel.
      Add a new AM HTTP endpoint for getting the container location information for external tools.
      Update Samza Dashboard and any other tools that rely on locality information to get it from the new endpoint instead of the JobModel.

      Attachments

        Issue Links

          Activity

            People

              bharathkk Bharath Kumarasubramanian
              pmaheshwari Prateek Maheshwari
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7.5h
                  7.5h