API Migration

[uraniumanchor]

This repo doesn't look terribly active, but while checking server logs I noticed that it's hitting api/v1/hosts. V1 as a whole and hosts in particular are going away in the near future. If there's anything that you need that isn't being provided by V2 please let me know.

[daenney]

Hi @uraniumanchor, thanks for reaching out! When I initially built the bot, and its library, @daenney/gdq, the V2 API seemed to exist but not really be functional. I popped by on the issue tracker at the time: GamesDoneQuick/donation-tracker#357

If v2 is up and running and is what's expected to be used now, I'm happy to make the changes. It would be great if there was some kind of API documentation, or if there's some way to get Django to output routes and expected message body formats to make this easier. I seem to have managed to get Django routes last time, but I don't exactly recall how I did it 😅.

[uraniumanchor]

I'm finishing up adding the Run model to the API today, and the rest should be following shortly. My current plan is to leave v1 around for a short while in read-only mode since we still have some stuff even on our end that's still using it to display things, but if all goes well v1 will be gone either before or shortly after AGDQ25.

As for snapshots and other documentation, I'm hoping to publish some example snapshots soon, generated from the testing framework. I'm also hoping to provide some OpenAPI specifications but automatically generating them has been not as simple as I'd hoped.

[daenney]

Generating the specs is very hit-and-miss, yeah. I hadn't thought of looking at the tests, but that's a good idea. I imagine I can also peak at some of the frontend code since that should be calling the tracker API too?

[daenney]

@uraniumanchor I've been doing the migration in the GDQ library, and so far so good. The new v2 API is much nicer, and thanks to /runs also containing the expanded talent objects I no longer have to query additional endpoints to bring that data together. This is really nice, so thank you for doing that.

The one bit where I've hit a snag is /interviews, since I need those to compute the full schedule. The problem I'm having is that the interviewers and subjects fields on the interview model are a string (in the JSON), instead of a list of talent. They seem to be joined by either & or , which makes them hard to parse into something regular. If those two fields became lists of talent like commentators, hosts and runners, that would be easier to work with. (I obviously don't expect this to change since I'm just one weirdo on the internet, so in the mean time I'll do the parsing.)

The other thing I noticed on interviews is that subjects isn't always a talent/person etc. For example for AGDQ 2021 I see things like "subjects": "Prizes!?". Which feels more like what should be in the topic, but that one has "topic": "Okay. Not you in the studio though, the general you". So I'm a little unsure as to what to do with those.

[daenney]

Oh, one other thing; the old event model used to have a total donation count. I'm not sure how to get to that one now. It was useful so I could have the bot announce things like whenever we crossed a new donation threshold (every 10k for example).

[uraniumanchor]

I do plan on migrating those Interview fields to a list of talent, but that won't be changed until after AGDQ at the earliest. For now they're still just freeform text fields.

subjects is INTENDED to be a list of people (it used to be called interviewees) but since it's currently a freeform field people have just been sorta putting whatever into it. topic is and shall remain freeform field as its supposed to be indicating if it's a charity interview, an interview about a past or upcoming run, etc. I think your particular example was somebody trying to be cute.

As for your last question, if you include ?totals in the query string, the event query should still return the total/count aggregates as amount and donation_count. I'm considering making this only work if you ask for a specific event, though, as it's sort of expensive on the DB side. Either that or caching the results somewhere.

[daenney]

I do plan on migrating those Interview fields to a list of talent, but that won't be changed until after AGDQ at the earliest. For now they're still just freeform text fields.

Good to know! Then I'll leave interviews out for now, so I don't have to change the API my library exposes on people later on. It never used to include interviews in the schedule so that's fine for its current usage.

subjects is INTENDED to be a list of people (it used to be called interviewees) but since it's currently a freeform field people have just been sorta putting whatever into it. topic is and shall remain freeform field as its supposed to be indicating if it's a charity interview, an interview about a past or upcoming run, etc. I think your particular example was somebody trying to be cute.

Makes sense.

As for your last question, if you include ?totals in the query string, the event query should still return the total/count aggregates as amount and donation_count. I'm considering making this only work if you ask for a specific event, though, as it's sort of expensive on the DB side. Either that or caching the results somewhere.

Ah awesome, I hadn't realised that. I only ever query it for a specific event. The only time I query /events/ without and ID is to get a list of events itself, but at that point I only care about the id, name, short and datetime.

[daenney]

One thing I was thinking about for the donations; you could store the count and total on the event, and recalculate it in the background from time to time?

Once an event is finished, those numbers never update, so including them in /events or /events/ID is essentially free. For ongoing events you could have a Celery job that recalculates those values every X minutes, with the caveat that the value would be potentially stale for that time period. But maybe that's OK if it's not used for live stats anywhere.

[uraniumanchor]

Internally I'm already storing DonorCache for similar purposes so I could most likely adapt it to store event totals as well. I'll just have to check that the performance is still acceptable for that. Certainly would work for older, locked events, it's just the live events I'm concerned about for that particular method.