Hey @nisrael and fr-tech folks, this is done pending @nisrael's review.

Adding @JMando : Could you please give us a description of how the landing pages should be parameterized so that the data can be easily assessed for this project? Note that there is a seconardy request to determine the split of mobile vs desktop devices on that land on the TY page, though that is not necessary if it adds to the lift.

Thanks @KHaggard : I'd suggest email_greeting. In the UI, it is listed as Email Greeting and in the database it's listed as email_greeting_display so for the export, email_greeting seems like a human-readable term that wouldn't get confused between UI and backend.

Thank you, and thanks to Thea! We do want to keep in coordination with data and import / export requests, but since this one involves obvious PII, we (Analytics) won't have an active role. We don't capture this information on our side. I'll keep an eye on the ticket for progress, and much appreciate the info and being looped in on the request.

Great @Ottomata yes it is in that yaml file!

Thank you @Jdrewniak and @Ottomata! This is a schema that I've worked with before to have the data whitelisted for permanent data capture. Is there anything I'll need to do on my end to ensure that this process continues after the migration?

Hi @Mayakp.wiki, I can add some context here. Below are the campaign dates for the English fundraiser for the last 3 years, though we also want to highlight that it's not our highest priority going into next week to provide exact breakdowns for those days. As we've seen declines in all KPIs for all days of the campaign YoY(oY), the day of year / day of month / day of week between years would be interesting to see but not of urgent need.

Awesome thank you @mforns! @JMando and I can tackle this over the next few weeks.

Hi all! Pinging @mforns who was helpful with the original whitelist request from phab ticket T273246. I recall that we had some discussion over whether we could keep the referrer data in the sanitized version of this data, which is why we created this new task - to scrub that data.

Group ID = 310 - noting for future

@DStrine to confirm, is this a static Civi group?

Hey @Jgreen yes this will need to be on fran1001 - correct. Thanks for the note on python3 - we will keep using that path!

@DStrine Let's plan to talk about it during out next FR-Tech / Analytics sync up - I think there are nuances here that will be better to talk through. I'll put it on the agenda. Thanks!

@DStrine this task is to look at these elements from the database side - where is this data physically collected on the database and how often do we get it? I don't think it should involve the email team. We know the workflows exist - this task is just to get aligned between FR-Tech and Analytics about the data capture.

A temporary inconvenience @BTullis , thank you for the quick fix! I am back up and running and I'll let @JMando confirm when he gets some time. I can now see the SQL Lab drop down menu as well as run queries.

Reading @odimitrijevic's comment, I'm going to proactively add @JMando to ask that he also have SQL Lab enabled (I will also add this to the description). Thanks!

Thank you, @Ejegg - that is interesting! I'm adding a comment here from a donor ticket, just in case this is helpful and related to this same issue. This was ticket #1071828 in case you have access to / can reference ZD, and per a Slack thread I had about this with others, the issue is "once I select my donation amount and use Apple Pay to check out, nothing happens. There is no further checkout process or submission." Sounds like it might be related, so I wanted to relay this here; as others have noted, it could be completely unrelated or a one-off issue as well.

Here is a query to show the donations which have come in through utm_mediums of 'endowment' but have a financial type of 9 (cash):

Amazing! @razzi Thank you so much! I was trying at this all day and I think what the difference was that it would not respond to my email (eyener@) for recovery, but only to my old contractor email (eyenever-ctr@) Well, something in the docs triggered it so thank you very much. I've logged into both Superset and Gerrit. Closing this out and thank you again!!

Hi all, to circle around from the conversation via email, @JMando and I can take this on. Thanks!

Marking resolved as @BTullis was able to help us get this working. Thank you!

Thank you @BTullis ! Apologies for missing the ownership of event_sanitized. - I tried to edit this just now and it is working. I also tried to add @JMando via his alias (which ended up being just jm ) and was able to do so via the legacy editor. Thank you - we're all set now. I'll mark this resolved as well as the other open task we have.

Hi @BTullis thanks - looking back at this task, it is actually the dataset currently owned by @mpopov (schema: event_sanitized) that I would need to be added as owner of.

Thanks @BTullis - we do use Superset's SQL Lab, which is okay at showing a schema once you know the dataset you want to query, but not all we would like a SQL UI to be in terms of showing the whole of available databases and their tables. We can use manual commands in beeline to accomplish this, but a UI is very helpful.

So interesting! Thank you @BTullis, adding @Dwisehaupt to this thread because we were having this same issue - not seeing full user list in dropdown to add them to the dataset owner list - in the Fundraising instance of Superset. @Dwisehaupt looks like I'll just have to use the legacy version until 1.4.0+ is released. Thanks!

That's great to know @odimitrijevic ! We can work with this data in Superset, yes.

Ah sorry, @Eileenmcnaughton can I help clarify more? In broad terms, it would be great if we could create a searchkit and then see a big aggregated total of how many contacts / activities / etc the searchkit contains : There are 14,000 donors in this searchkit (and maybe drill down into that 14,000 to bring up a list of donors).

Declining. Previously, the silverpop tables were used in some of our scripts for an extra layer of dedupe, but (1) dedupe has evolved since this ticket creation and (2) the effort involved by all parties is much higher than the reward to add that added layer of dedupe on raw email addresses.

Thanks for taking a look @odimitrijevic! Here is one where both @JMando and I received the Results have expired, rerun the query if needed. error in Hue last Friday:

Interestingly, I don't seem to get this error using the pivot table chart type - this could be isolated to the table chart type

Thank you! I'm adding @JMando here as a subscriber. We have a few metrics we'd like to add to the event_sanitized.centralnoticebannerhistory table in Superset and it would be great if we both had access to modify this data source.

Added contact info for @JLaytonWMF to the Collab page.

Of course, @cchen!

We should collaborate after code freeze in Q3 with @Cstone to turn logfiles into more usable raw files. Can we follow up by creating a new task on that as a first step? Thank you!

Thanks, @Eileenmcnaughton this is great! I'm seeing them in wmf_donor now.

Great! Thank you. I'll be there for the second half of it (I have a conflict for the first 30 minutes).

Hey XenoRyet checking in - would Tuesday or Wednesday this week be a good time to bring this to Tech Talks?

Thanks, @Ejegg, I haven't read through the comment thread thoroughly, but due to this being authored in 2016 and now having much better (entirely new?) systems of tracking this, I'd say we can close this and start afresh. We have most of this through other systems, and I'd be happy to talk to any stakeholders on the ticket + FR-Tech to see if / what is outstanding.

Hi @Ottomata! Actually that would be super helpful. Would you mind picking anything on my calendar that is open and works for you? I'll remove unnecessary events, and I have my working hours blocked off.

Hi @Ottomata thanks for the ping - this is on my list of quarterly projects, and I've scheduled time out of this week to focus on it in earnest. I've read through the docs and I'll start on step 3 as outlined here this week. I might be pinging back here or on IRC if I run into any questions!

Hi all, just piping in to say thank you for adding this to your queue! There is no urgent need to add this to this quarter's work (and wow, this quarter is nearly done with) but I look forward to using this data when it becomes available.

Ah, nice. I'll go get my vision checked and close out this task. That you for correcting my numerous typos during this setup process.

Hi again! One further question for you all; does @JMando have access to jupyter? The command ssh -N stat1005.eqiad.wmnet -L 8880:127.0.0.1:8880 seems to open a tunnel successfully but he was unable to reach Jupyter at localhost:8000 ("This site can't be reached" error).

Oh yes. That will help. :) Connected, thank you!

Thanks! Error for me when I ssh -v stat1007.eqiad.wmnet:

Hi all! Thank you for working on this and granting access for @JMando! We have been working from the level of access authorized, which you've all granted, to trying to confirm that we can login as expected, and hitting some roadblocks - probably user error on my part, as I'm guessing the instructions I've written up have changed since I last checked.

Yeah I agree with @DStrine. We have a request in for all_funds_last_donated and similar all_funds categories to be implemented that this would fall under.

Thank you for the notes, and absolutely approved! @JMando is a 40-hour / week. long-term contractor in a senior level position with a full confidentiality agreement who is a core member of our team. He will need access to production data on stat100x machines in order to query, model, and analyze data that impacts Fundraising Analytics outcomes, such as pageviews and banner history event collection during Fundraising campaigns.

Adding @JMando to keep track of changes. @Eileenmcnaughton just so I understand it: the current state is that the FY2122 fields are added and in production in the current table (we can see them, and thank you!) but the new fields and new indexing are going to be covered by this ticket?

Chiming in here to clarify a few things on my side - are the 'both_' fields specific to Acoustic? Are they already stored somewhere under this name? If so, what type of effort would be involved in renaming?

Pinging this thread again, as this has come up as an Analytics request to report on RML data:

@Eileenmcnaughton and I talked about this today. In addition to the FY2122 rollup field, I would also be looking for:

• foundation_total_fy1819
• foundation_total_fy1920
• foundation_total_fy2021
• foundation_total_fy2122
• foundation_largest_donation
• foundation_number_donations
• foundation_first_donation_date
• foundation_first_donation_amount
• foundation_last_donated
• foundation_lifetime_usd_total
• foundation_date_of_largest_donation
• change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - AF only)
• endowment_change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - Endowment only)
• foundation_change_fy2021_fy2122 (or the equivalent change between FY2021 and FY2122 metric - combined funds)
• foundation_change_fy1920_fy2021 (or the equivalent change between FY1920 and FY2021 metric - combined funds)
• foundation_change_fy1819_fy1920 (or the equivalent change between FY1819 and FY1920 metric - combined funds)

Update on the immediate use case for this that I was able to pretty easily provide an update for the Email team on who was / was not on the MSL on their necessary list.

@Dwisehaupt Just noting that I'll send Joseph your way today once he's in his Gmail and IRC, etc, to get setup configured. I forgot to mention that he won't have the 'usual' superset permissions but he'll need an Admin account instead of a Data Consumer account. Thanks!

Adding to the list that Joseph will also need a Jupyter instance. I could create a separate task for that if this isn't onboarding-appropriate yet.

Oh, Joseph also has an ITS-provided yubikey and macbook pro laptop.

Joseph's start date: July 19, 2021. Anything we can get access to (like sql / server / Superset / SSH access setup started) would be awesome! Thank you!

Ah, got it. Thanks @Eileenmcnaughton

Makes perfect sense, there are a lot of systems that these would have to go live in. I would just be interested in involvement up to the point of seeing new fields visible in civicrm.wmf_donor - we would use these as soon as they're available (my fault I didn't double check that FY2122 was available already).

Just guessing - probably for the Civi UI update to ensure these fields are visible in the Civi UI, but for use in analytics and reporting, the update to the civicrm.wmf_donor table will get us where we need to be for most use cases, where we can work with this for the Japan campaign and ongoing for 99% of reporting

Hi @Eileenmcnaughton - I was just asking about this today! I'll offer to write a commit for wmf_donor for this and submit it to you as a reviewer, if @DStrine is okay with that, as we use these fields all the time in Analytics. I would also like to propose adding a few additional fields that combine Endowment and Annual Fund totals, as this would cut down on the post-processing I do in our data cubes.

Sorry @Eileenmcnaughton I'm coming back to this now; is the query in prod in the correct format, or was the patch to make the UI more SQL-like, or something different...? Thanks for the help! Either way, it's working now.

@Eileenmcnaughton thanks! This is working in dev but not in prod. Here is the same searchkit reproduced in prod. I'm getting a spinning wheel when I click on 'search' and the results are unexpected: CIDs as old as 1 are showing up in the initial results, which never fully return.

Just a note as I saw this pop up - can we change the task title to New FY2021 Donors to avoid confusion? I think I understand from context that this is to be used in FY2122, but my two cents is that it will be hard to tell in the future what links these donors to FY2122. Thanks!

I'm trying to test this under the Mailing Datas test searchkit and not seeing any results come up when query for 3 mailing_identifiers that were sent out around October - November

The below query pulls a total of 832,108 donors. @LeanneS I realize this is lower than expected: there are a number of US individuals who are first time donors in FY2021 who are opted in (criteria mentioned above) who nonetheless do not have a listed street address, mostly due to matching gifts (at first glance).

Thank you! Yes we can get this done in Q1.

@LeanneS where should the data pull live on the file server once I gather the data? I'll be working on this tomorrow.

Fair enough, @Ottomata! Is Q1 too late for this self-imposed deadline?

@Ottomata when would this migration need to happen? If we have a deadline, we might be able to prioritize it and help find an owner.

Thanks @JKatzWMF - I'm seeing the mobile subdomains now! And thanks @Pcoombe for all your help here.