Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yieldlab: Add Digital Service Act (DSA) support #3011

Merged
merged 7 commits into from
Mar 26, 2024
+322 −3
Merged

Yieldlab: Add Digital Service Act (DSA) support #3011

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cc0cca6
Yieldlab: Implementing DSA support
Feb 20, 2024
edf9d9f
Code review changes
Feb 26, 2024
3c57c36
Addressing code review
Mar 4, 2024
35f54c5
Addressing requested changes
Mar 5, 2024
e07cb0b
Addressing more requested changes
Mar 7, 2024
0ffb3a7
Removing redundant null check
Mar 11, 2024
358acf0
Addressing more requested changes
Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 84 additions & 1 deletion src/main/java/org/prebid/server/bidder/yieldlab/YieldlabBidder.java
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comparing the logic with the GO version, I don't see any sorting operations for adSlots in GO. Does PBS-Java need this?
adSlotIds.stream().sorted().collect(...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't needed, no.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, i don't see anything similar in GO for ts parameter:
isDebugEnabled(request) ? "200000" : ...

Does PBS-Java need to pass this debug value? Just out of interest I removed the debug part and all tests passed:
final String timestamp = String.valueOf(clock.instant().getEpochSecond());

Same question about
final int weekNumber = isDebugEnabled(bidRequest) ? 35 : ...

and

        final String timestamp = isDebugEnabled(bidRequest)
                ? "200000"
                : ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as we are concerned, it is not needed no.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String.join("ylid:", user.getBuyeruid()) doesn't corresponds to GO logic. It just doesn't do anything.
It's need to be "ylid" + user.getBuyeruid().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a courtesy I can fix this, but this wasn't written by us and is also out of scope.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementing this breaks the tests constructExtImpShouldWorkWithDuplicateKeysTargeting and makeHttpRequestsShouldSendRequestToModifiedUrlWithHeaders, so I will have to decline.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

                uriBuilder.addParameter("lat", resolveNumberParameter(geo.getLat()));
                uriBuilder.addParameter("lon", resolveNumberParameter(geo.getLon()));

This code doesn't handle nulls as GO does.
Java: null -> null
GO: nil -> "0"

Copy link
Contributor Author

@william-harris william-harris Mar 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is out of scope, and I don't intend to change it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I don't see implementation of this part:

	if hasFormats, formats := a.makeFormats(req); hasFormats {
		q.Set("sizes", formats)
	}

Will it be added later?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I don't see implementation of this part:

	if req.Source != nil && req.Source.Ext != nil {
		if openRtbSchain := unmarshalSupplyChain(req); openRtbSchain != nil {
			if schainValue := makeSupplyChain(*openRtbSchain); schainValue != "" {
				q.Set("schain", schainValue)
			}
		}
	}

Will it be added later?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope, but....

Can i ask you to extract collectImpExt(imps) from ExtImpYieldlab getMatchedExtImp(...) method, so it will be out of for loop on bids? Just to make it from O(nk) to O(n).... 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but I must decline. Like you said it is out of scope.

Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.iab.openrtb.request.App;
import com.iab.openrtb.request.BidRequest;
import com.iab.openrtb.request.Device;
Expand All @@ -23,11 +25,15 @@
import org.prebid.server.bidder.model.BidderError;
import org.prebid.server.bidder.model.HttpRequest;
import org.prebid.server.bidder.model.Result;
import org.prebid.server.bidder.yieldlab.model.YieldlabDigitalServicesActResponse;
import org.prebid.server.bidder.yieldlab.model.YieldlabResponse;
import org.prebid.server.exception.PreBidException;
import org.prebid.server.json.DecodeException;
import org.prebid.server.json.JacksonMapper;
import org.prebid.server.proto.openrtb.ext.ExtPrebid;
import org.prebid.server.proto.openrtb.ext.request.ExtRegs;
import org.prebid.server.proto.openrtb.ext.request.ExtRegsDsa;
import org.prebid.server.proto.openrtb.ext.request.ExtRegsDsaTransparency;
import org.prebid.server.proto.openrtb.ext.request.ExtRequest;
import org.prebid.server.proto.openrtb.ext.request.ExtRequestPrebid;
import org.prebid.server.proto.openrtb.ext.request.ExtUser;
Expand All @@ -41,9 +47,11 @@
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.stream.Collectors;

public class YieldlabBidder implements Bidder<Void> {
Expand All @@ -58,6 +66,9 @@ public class YieldlabBidder implements Bidder<Void> {
private static final String CREATIVE_ID = "%s%s%s";
private static final String AD_SOURCE_BANNER = "<script src=\"%s\"></script>";
private static final String AD_SOURCE_URL = "https://ad.yieldlab.net/d/%s/%s/%s?%s";
private static final String TRANSPARENCY_TEMPLATE = "%s~%s";
private static final String TRANSPARENCY_TEMPLATE_PARAMS_DELIMITER = "_";
private static final String TRANSPARENCY_TEMPLATE_DELIMITER = "~~";
private static final String VAST_MARKUP = """
<VAST version="2.0"><Ad id="%s"><Wrapper>
<AdSystem>Yieldlab</AdSystem>
Expand Down Expand Up @@ -189,6 +200,8 @@ private String makeUrl(ExtImpYieldlab extImpYieldlab, BidRequest request) {
uriBuilder.addParameter("consent", consent);
}

extractDsaRequestParamsFromBidRequest(request).forEach(uriBuilder::addParameter);

return uriBuilder.toString();
}

Expand Down Expand Up @@ -231,6 +244,60 @@ private static String getConsentParameter(User user) {
return ObjectUtils.defaultIfNull(consent, "");
}

private static Map<String, String> extractDsaRequestParamsFromBidRequest(BidRequest request) {
return Optional.ofNullable(request.getRegs())
.map(Regs::getExt)
.map(ExtRegs::getDsa)
.map(YieldlabBidder::extractDsaRequestParamsFromDsaRegsExtension)
.orElseGet(Map::of);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are preferring to use Collections.empty* instead of *.of().
Also, as Collections.emptyMap() (and Map.of()) returns already created object, it's better to use .orElse instead of orElseGet.

So, please, change it with:
.orElse(Collections.emptyMap());

}

private static Map<String, String> extractDsaRequestParamsFromDsaRegsExtension(final ExtRegsDsa dsa) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant final keyword for method parameter. Please, remove it.

final HashMap<String, String> dsaRequestParams = new HashMap<>();
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved

if (dsa.getDsaRequired() != null) {
dsaRequestParams.put("dsarequired", dsa.getDsaRequired().toString());
}

if (dsa.getPubRender() != null) {
dsaRequestParams.put("dsapubrender", dsa.getPubRender().toString());
}

if (dsa.getDataToPub() != null) {
dsaRequestParams.put("dsadatatopub", dsa.getDataToPub().toString());
}
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved

final List<ExtRegsDsaTransparency> dsaTransparency = dsa.getTransparency();
if (dsaTransparency != null && !dsaTransparency.isEmpty()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Predicate can be simplified: CollectionUtils.isNotEmpty(dsaTransparency)

dsaRequestParams.put("dsatransparency", encodeTransparenciesAsString(dsaTransparency));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

encodeTransparencyAsString can return an empty string if all entries are invalid. PBS-GO doesn't add the dsatransparency parameter in this case.

}
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved

return dsaRequestParams;
}

private static String encodeTransparenciesAsString(List<ExtRegsDsaTransparency> transparencies) {
return transparencies.stream()
.filter(YieldlabBidder::transparencyIsValid)
.map(YieldlabBidder::encodeTransparency)
.collect(Collectors.joining(TRANSPARENCY_TEMPLATE_DELIMITER));
}

private static boolean transparencyIsValid(ExtRegsDsaTransparency transparency) {
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved
return transparency.getDomain() != null
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved
&& transparency.getDsaParams() != null
&& !transparency.getDsaParams().isEmpty();
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved
}

private static String encodeTransparency(ExtRegsDsaTransparency transparency) {
return TRANSPARENCY_TEMPLATE.formatted(transparency.getDomain(),
encodeTransparencyParams(transparency.getDsaParams()));
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved
}

private static String encodeTransparencyParams(List<Integer> dsaParams) {
return dsaParams.stream().map(Objects::toString).collect(Collectors.joining(
TRANSPARENCY_TEMPLATE_PARAMS_DELIMITER));
}

private static MultiMap resolveHeaders(Site site, Device device, User user) {
final MultiMap headers = MultiMap.caseInsensitiveMultiMap()
.add(HttpUtil.ACCEPT_HEADER, HttpHeaderValues.APPLICATION_JSON);
Expand Down Expand Up @@ -339,7 +406,8 @@ private Bid.BidBuilder addBidParams(YieldlabResponse yieldlabResponse, BidReques
.dealid(resolveNumberParameter(yieldlabResponse.getPid()))
.crid(makeCreativeId(bidRequest, yieldlabResponse, matchedExtImp))
.w(resolveSizeParameter(yieldlabResponse.getAdSize(), true))
.h(resolveSizeParameter(yieldlabResponse.getAdSize(), false));
.h(resolveSizeParameter(yieldlabResponse.getAdSize(), false))
.ext(resolveExtParameter(yieldlabResponse.getDsa()));

return updatedBid;
}
Expand Down Expand Up @@ -408,4 +476,19 @@ private String makeNurl(BidRequest bidRequest, ExtImpYieldlab extImpYieldlab, Yi
yieldlabResponse.getAdSize(),
uriBuilder.toString().replace("?", ""));
}

private ObjectNode resolveExtParameter(YieldlabDigitalServicesActResponse dsa) {
if (dsa == null) {
return null;
}
final ObjectNode ext = mapper.mapper().createObjectNode();
final JsonNode dsaNode;
try {
dsaNode = mapper.mapper().convertValue(dsa, JsonNode.class);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mapper.mapper().valueToTree(dsa)

} catch (IllegalArgumentException e) {
throw new PreBidException("Failed to serialize DSA object", e);
Copy link
Collaborator

@AntoxaAntoxic AntoxaAntoxic Mar 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like the PBS Go version has a more detailed error message fmt.Errorf("failed to make JSON for seatbid.bid.ext for adslotID %v. This is most likely a programming issue", bid.ID)

Perhaps it's worth keeping it here as well

Also, according to the PBS Go version the error shouldn't disrupt other bids processing, so the error has to be reported and the ext has to be null in this case. Please double-check

}
ext.set("dsa", dsaNode);
return ext;
}
}
26 changes: 26 additions & 0 deletions ...main/java/org/prebid/server/bidder/yieldlab/model/YieldlabDigitalServicesActResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package org.prebid.server.bidder.yieldlab.model;

import lombok.AllArgsConstructor;
import lombok.Value;

import java.util.List;

@AllArgsConstructor(staticName = "of")
@Value(staticConstructor = "of")
public class YieldlabDigitalServicesActResponse {

String behalf;

String paid;

Integer adrender;

List<Transparency> transparency;

@AllArgsConstructor(staticName = "of")
@Value(staticConstructor = "of")
public static class Transparency {
String domain;
List<Integer> dsaparams;
}
}
2 changes: 2 additions & 0 deletions src/main/java/org/prebid/server/bidder/yieldlab/model/YieldlabResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,6 @@ public class YieldlabResponse {
Integer did;

String pvid;

YieldlabDigitalServicesActResponse dsa;
}
Loading
Loading