Page MenuHomePhabricator
Create Task
Maniphest T289224

Split search.wikimedia.org out of ops/mediawiki-config into separate service
Closed, ResolvedPublic
Actions

Description

search.wikimedia.org is the gateway for Apple's dictionary search. The entire thing is ~100 lines of PHP: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/docroot/search.wikimedia.org/index.php and purely just a wrapper around a public API.

It has no tests, logging, stats, etc. AFAICT. There is no real reason to keep deploying it with MediaWiki like this.

I propose that search.wikimedia.org is turned into a PHP microservice deployed via k8s and separated from MediaWiki. Putting it in a separate repository would it make it easy to add PHPUnit tests and deployment pipeline. Deploying via k8s will give us logging and basic monitoring/stats via envoy.

I came across this via T288848: Make HTTP calls work within mediawiki on kubernetes , where the proposed solution would not work for search.wikimedia.org, creating some extra work for MW-on-K8s.

  • Review charts
  • apple-search namespaces in k8s
  • apple-search accounts in k8s.
  • apple-search puppet private tokens.
  • Generate TLS certificates
  • Review helmfile.d files:
  • LVS setup
  • DNS for LVS records
  • Discovery DNS
  • Monitoring dashboard
  • Integration and Acceptance tests

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

Legoktm created this task.Aug 19 2021, 1:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2021, 1:24 AM
RhinosF1 subscribed.Aug 19 2021, 6:54 AM
Gehel added a project: Discovery-Search.Aug 19 2021, 6:55 AM
taavi subscribed.Aug 19 2021, 6:57 AM
Urbanecm moved this task from Backlog to Analysis / under discussion on the Wikimedia-Site-requests board.Aug 19 2021, 1:53 PM
Peachey88 subscribed.Aug 19 2021, 7:37 PM
taavi added a comment.Aug 22 2021, 6:21 PM

+1 to moving this out of ops/mw-config. I'm happy to try to get this moving if the stakeholders (Search team I assume) are fine with it.

Gehel removed projects: Discovery-Search, Discovery-ARCHIVED.Aug 23 2021, 3:14 PM
Gehel subscribed.

The Search Platform team isn't responsible for this service, so from my point of view, feel free to do whatever needs doing!

Legoktm added a comment.Aug 23 2021, 5:18 PM
In T289224#7301754, @Gehel wrote:

The Search Platform team isn't responsible for this service, so from my point of view, feel free to do whatever needs doing!

This is not what I was expecting, but thanks for the comment. I think we'll go ahead with the split and migration to k8s and I'll file a separate Code-Stewardship-Reviews for it.

In T289224#7299972, @Majavah wrote:

+1 to moving this out of ops/mw-config. I'm happy to try to get this moving if the stakeholders (Search team I assume) are fine with it.

Please go ahead :) Originally on IRC I had suggested a repository name of search/apple-gateway but that was assuming Search was responsible for it. I think we can use something like: mediawiki/services/apple-search (bonus points if you can use git filter-branch etc. to keep the VCS history!).

I think you/we should be able to copy most of the setup from Shellbox, so the architecture is envoy (TLS termination) -> httpd (apache) ->php-fpm -> 100 line PHP script.

taavi claimed this task.Aug 23 2021, 5:29 PM
Restricted Application added a project: User-Majavah. · View Herald TranscriptAug 23 2021, 5:29 PM
Legoktm added a comment.EditedAug 23 2021, 5:40 PM

In the 1/128 sampled web request logs, this service has 1.7k entries over 30 days, so theoretically ~217,600 reqs/month, or ~5reqs/min. For reference in 2013, we were getting 11 reqs/second (per T81982). It really should be fine with 2 replicas and low memory/CPU limits.

Legoktm mentioned this in T289518: Code Stewardship Review for search.wikimedia.org.Aug 23 2021, 5:51 PM
Legoktm renamed this task from Consider splitting search.wikimedia.org out of ops/mediawiki-config into separate service to Split search.wikimedia.org out of ops/mediawiki-config into separate service.Aug 23 2021, 6:23 PM
Jdforrester-WMF subscribed.Aug 23 2021, 9:03 PM
gerritbot added a comment.Aug 28 2021, 7:17 PM

Change 715317 had a related patch set uploaded (by Majavah; author: Majavah):

[integration/config@master] Setup CI for mediawiki/services/apple-search

https://gerrit.wikimedia.org/r/715317

gerritbot added a project: Patch-For-Review.Aug 28 2021, 7:17 PM
gerritbot added a comment.Aug 30 2021, 3:15 PM

Change 715317 merged by jenkins-bot:

[integration/config@master] jjb, Zuul: [mediawiki/services/apple-search] Add pipeline CI

https://gerrit.wikimedia.org/r/715317

Stashbot added a comment.Aug 30 2021, 3:16 PM

Mentioned in SAL (#wikimedia-releng) [2021-08-30T15:16:45Z] <James_F> Zuul: [mediawiki/services/apple-search] Add pipeline CI for T289224

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2021, 4:10 PM
Jdforrester-WMF added a comment.Aug 30 2021, 7:25 PM

https://integration.wikimedia.org/ci/job/apple-search-pipeline-publish/1/console:

'publishedImage':'docker-registry.wikimedia.org/wikimedia/mediawiki-services-apple-search:2021-08-30-191349-production'

Legoktm added a project: Service-deployment-requests.Sep 1 2021, 9:35 PM
Legoktm updated the task description. (Show Details)
Restricted Application added projects: Services, SRE. · View Herald TranscriptSep 1 2021, 9:35 PM
gerritbot added a comment.Sep 1 2021, 9:59 PM

Change 716049 had a related patch set uploaded (by Legoktm; author: Legoktm):

[labs/private@master] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716049

gerritbot added a project: Patch-For-Review.Sep 1 2021, 9:59 PM
gerritbot added a comment.Sep 1 2021, 10:03 PM

Change 716052 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716052

gerritbot added a comment.Sep 1 2021, 10:16 PM

Change 716049 merged by Legoktm:

[labs/private@master] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716049

gerritbot added a comment.Sep 1 2021, 10:16 PM

Change 716052 merged by Legoktm:

[operations/puppet@production] Add k8s users/tokens for apple-search

https://gerrit.wikimedia.org/r/716052

gerritbot added a comment.Sep 1 2021, 10:21 PM

Change 716056 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] admin: Add namespace for apple-search

https://gerrit.wikimedia.org/r/716056

Legoktm mentioned this in rLPRI84cbaa4eb529: Add k8s users/tokens for apple-search.Sep 1 2021, 10:24 PM
gerritbot added a comment.Sep 1 2021, 10:39 PM

Change 716056 merged by jenkins-bot:

[operations/deployment-charts@master] admin: Add namespace for apple-search

https://gerrit.wikimedia.org/r/716056

Legoktm updated the task description. (Show Details)Sep 1 2021, 11:06 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 1 2021, 11:10 PM
gerritbot added a comment.Oct 6 2021, 6:12 PM

Change 726933 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/deployment-charts@master] apple-search: New chart

https://gerrit.wikimedia.org/r/726933

gerritbot added a project: Patch-For-Review.Oct 6 2021, 6:12 PM
gerritbot added a comment.Oct 21 2021, 6:34 AM

Change 726933 merged by jenkins-bot:

[operations/deployment-charts@master] apple-search: New chart

https://gerrit.wikimedia.org/r/726933

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2021, 7:10 AM
gerritbot added a comment.Nov 2 2021, 4:24 PM

Change 736273 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] Add apple-search deployment

https://gerrit.wikimedia.org/r/736273

gerritbot added a project: Patch-For-Review.Nov 2 2021, 4:24 PM
gerritbot added a comment.Nov 22 2021, 8:10 AM

Change 736273 merged by jenkins-bot:

[operations/deployment-charts@master] Add apple-search deployment

https://gerrit.wikimedia.org/r/736273

gerritbot added a comment.Nov 22 2021, 2:57 PM

Change 740598 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] service::catalog: add apple-search

https://gerrit.wikimedia.org/r/740598

gerritbot added a comment.Nov 22 2021, 2:57 PM

Change 740599 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] apple-search: move to lvs setup

https://gerrit.wikimedia.org/r/740599

gerritbot added a comment.Nov 22 2021, 2:57 PM

Change 740600 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] apple-search: enable monitoring

https://gerrit.wikimedia.org/r/740600

gerritbot added a comment.Nov 22 2021, 2:57 PM

Change 740601 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] apple-search: promote to production

https://gerrit.wikimedia.org/r/740601

gerritbot added a comment.Nov 22 2021, 3:06 PM

Change 740598 merged by Giuseppe Lavagetto:

[operations/puppet@production] service::catalog: add apple-search

https://gerrit.wikimedia.org/r/740598

gerritbot added a comment.Nov 22 2021, 3:11 PM

Change 740599 merged by Giuseppe Lavagetto:

[operations/puppet@production] apple-search: move to lvs setup

https://gerrit.wikimedia.org/r/740599

gerritbot added a comment.Nov 22 2021, 3:49 PM

Change 740600 merged by Giuseppe Lavagetto:

[operations/puppet@production] apple-search: enable monitoring

https://gerrit.wikimedia.org/r/740600

gerritbot added a comment.Nov 22 2021, 4:04 PM

Change 740601 merged by Giuseppe Lavagetto:

[operations/puppet@production] apple-search: promote to production

https://gerrit.wikimedia.org/r/740601

gerritbot added a comment.Nov 22 2021, 5:08 PM

Change 740642 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Route search.wm.o to apple-search

https://gerrit.wikimedia.org/r/740642

gerritbot added a comment.Nov 23 2021, 7:31 AM

Change 740763 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver: rule for search.wikimedia.org

https://gerrit.wikimedia.org/r/740763

gerritbot added a comment.Nov 24 2021, 8:11 AM

Change 740763 merged by Giuseppe Lavagetto:

[operations/puppet@production] trafficserver: rule for search.wikimedia.org

https://gerrit.wikimedia.org/r/740763

gerritbot added a comment.Nov 24 2021, 8:16 AM

Change 741079 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] Remove search.wikimedia.org from appservers

https://gerrit.wikimedia.org/r/741079

gerritbot added a comment.Nov 24 2021, 11:26 AM

Change 741115 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/mediawiki-config@master] Remove search.wikimedia.org files

https://gerrit.wikimedia.org/r/741115

gerritbot added a comment.Nov 24 2021, 3:51 PM

Change 741079 merged by Giuseppe Lavagetto:

[operations/puppet@production] Remove search.wikimedia.org from appservers

https://gerrit.wikimedia.org/r/741079

gerritbot added a comment.Nov 24 2021, 5:54 PM

Change 740642 abandoned by Majavah:

[operations/puppet@production] hieradata: Route search.wm.o to apple-search

Reason:

already done in another patch

https://gerrit.wikimedia.org/r/740642

Legoktm awarded a token.Nov 24 2021, 7:57 PM
gerritbot added a comment.Nov 29 2021, 12:04 PM

Change 741115 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove search.wikimedia.org files

https://gerrit.wikimedia.org/r/741115

taavi closed this task as Resolved.Nov 29 2021, 12:07 PM
Stashbot added a comment.Nov 29 2021, 12:07 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-29T12:07:27Z] <urbanecm@deploy1002> Synchronized docroot/: 4662224229cb4083b8b01de436ccd65e8c00e7dd: Remove search.wikimedia.org files (T289224) (duration: 00m 56s)

mpopov mentioned this in T316296: Sunset search.wikimedia.org service.Aug 25 2022, 9:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits