Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yieldlab: Add Digital Service Act (DSA) support #3011

Merged
merged 7 commits into from
Mar 26, 2024
+322 −3
Merged

Yieldlab: Add Digital Service Act (DSA) support #3011

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cc0cca6
Yieldlab: Implementing DSA support
Feb 20, 2024
edf9d9f
Code review changes
Feb 26, 2024
3c57c36
Addressing code review
Mar 4, 2024
35f54c5
Addressing requested changes
Mar 5, 2024
e07cb0b
Addressing more requested changes
Mar 7, 2024
0ffb3a7
Removing redundant null check
Mar 11, 2024
358acf0
Addressing more requested changes
Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 93 additions & 1 deletion src/main/java/org/prebid/server/bidder/yieldlab/YieldlabBidder.java
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comparing the logic with the GO version, I don't see any sorting operations for adSlots in GO. Does PBS-Java need this?
adSlotIds.stream().sorted().collect(...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't needed, no.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, i don't see anything similar in GO for ts parameter:
isDebugEnabled(request) ? "200000" : ...

Does PBS-Java need to pass this debug value? Just out of interest I removed the debug part and all tests passed:
final String timestamp = String.valueOf(clock.instant().getEpochSecond());

Same question about
final int weekNumber = isDebugEnabled(bidRequest) ? 35 : ...

and

        final String timestamp = isDebugEnabled(bidRequest)
                ? "200000"
                : ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as we are concerned, it is not needed no.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String.join("ylid:", user.getBuyeruid()) doesn't corresponds to GO logic. It just doesn't do anything.
It's need to be "ylid" + user.getBuyeruid().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a courtesy I can fix this, but this wasn't written by us and is also out of scope.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementing this breaks the tests constructExtImpShouldWorkWithDuplicateKeysTargeting and makeHttpRequestsShouldSendRequestToModifiedUrlWithHeaders, so I will have to decline.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

                uriBuilder.addParameter("lat", resolveNumberParameter(geo.getLat()));
                uriBuilder.addParameter("lon", resolveNumberParameter(geo.getLon()));

This code doesn't handle nulls as GO does.
Java: null -> null
GO: nil -> "0"

Copy link
Contributor Author

@william-harris william-harris Mar 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is out of scope, and I don't intend to change it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I don't see implementation of this part:

	if hasFormats, formats := a.makeFormats(req); hasFormats {
		q.Set("sizes", formats)
	}

Will it be added later?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I don't see implementation of this part:

	if req.Source != nil && req.Source.Ext != nil {
		if openRtbSchain := unmarshalSupplyChain(req); openRtbSchain != nil {
			if schainValue := makeSupplyChain(*openRtbSchain); schainValue != "" {
				q.Set("schain", schainValue)
			}
		}
	}

Will it be added later?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of scope, but....

Can i ask you to extract collectImpExt(imps) from ExtImpYieldlab getMatchedExtImp(...) method, so it will be out of for loop on bids? Just to make it from O(nk) to O(n).... 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but I must decline. Like you said it is out of scope.

Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.iab.openrtb.request.App;
import com.iab.openrtb.request.BidRequest;
import com.iab.openrtb.request.Device;
Expand All @@ -14,6 +16,9 @@
import io.netty.handler.codec.http.HttpHeaderValues;
import io.vertx.core.MultiMap;
import io.vertx.core.http.HttpMethod;
import io.vertx.core.logging.Logger;
import io.vertx.core.logging.LoggerFactory;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.lang3.ObjectUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.client.utils.URIBuilder;
Expand All @@ -23,11 +28,15 @@
import org.prebid.server.bidder.model.BidderError;
import org.prebid.server.bidder.model.HttpRequest;
import org.prebid.server.bidder.model.Result;
import org.prebid.server.bidder.yieldlab.model.YieldlabDigitalServicesActResponse;
import org.prebid.server.bidder.yieldlab.model.YieldlabResponse;
import org.prebid.server.exception.PreBidException;
import org.prebid.server.json.DecodeException;
import org.prebid.server.json.JacksonMapper;
import org.prebid.server.proto.openrtb.ext.ExtPrebid;
import org.prebid.server.proto.openrtb.ext.request.ExtRegs;
import org.prebid.server.proto.openrtb.ext.request.ExtRegsDsa;
import org.prebid.server.proto.openrtb.ext.request.ExtRegsDsaTransparency;
import org.prebid.server.proto.openrtb.ext.request.ExtRequest;
import org.prebid.server.proto.openrtb.ext.request.ExtRequestPrebid;
import org.prebid.server.proto.openrtb.ext.request.ExtUser;
Expand All @@ -41,13 +50,16 @@
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.stream.Collectors;

public class YieldlabBidder implements Bidder<Void> {

private static final Logger logger = LoggerFactory.getLogger(YieldlabBidder.class);
private static final TypeReference<ExtPrebid<?, ExtImpYieldlab>> YIELDLAB_EXT_TYPE_REFERENCE =
new TypeReference<>() {
};
Expand All @@ -58,6 +70,9 @@ public class YieldlabBidder implements Bidder<Void> {
private static final String CREATIVE_ID = "%s%s%s";
private static final String AD_SOURCE_BANNER = "<script src=\"%s\"></script>";
private static final String AD_SOURCE_URL = "https://ad.yieldlab.net/d/%s/%s/%s?%s";
private static final String TRANSPARENCY_TEMPLATE = "%s~%s";
private static final String TRANSPARENCY_TEMPLATE_PARAMS_DELIMITER = "_";
private static final String TRANSPARENCY_TEMPLATE_DELIMITER = "~~";
private static final String VAST_MARKUP = """
<VAST version="2.0"><Ad id="%s"><Wrapper>
<AdSystem>Yieldlab</AdSystem>
Expand Down Expand Up @@ -189,6 +204,8 @@ private String makeUrl(ExtImpYieldlab extImpYieldlab, BidRequest request) {
uriBuilder.addParameter("consent", consent);
}

extractDsaRequestParamsFromBidRequest(request).forEach(uriBuilder::addParameter);

return uriBuilder.toString();
}

Expand Down Expand Up @@ -231,6 +248,63 @@ private static String getConsentParameter(User user) {
return ObjectUtils.defaultIfNull(consent, "");
}

private static Map<String, String> extractDsaRequestParamsFromBidRequest(BidRequest request) {
return Optional.ofNullable(request.getRegs())
.map(Regs::getExt)
.map(ExtRegs::getDsa)
.map(YieldlabBidder::extractDsaRequestParamsFromDsaRegsExtension)
.orElse(Collections.emptyMap());
}

private static Map<String, String> extractDsaRequestParamsFromDsaRegsExtension(final ExtRegsDsa dsa) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant final keyword for method parameter. Please, remove it.

final Map<String, String> dsaRequestParams = new HashMap<>();

if (dsa.getDsaRequired() != null) {
dsaRequestParams.put("dsarequired", dsa.getDsaRequired().toString());
}

if (dsa.getPubRender() != null) {
dsaRequestParams.put("dsapubrender", dsa.getPubRender().toString());
}

if (dsa.getDataToPub() != null) {
dsaRequestParams.put("dsadatatopub", dsa.getDataToPub().toString());
}
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved

final List<ExtRegsDsaTransparency> dsaTransparency = dsa.getTransparency();
if (CollectionUtils.isNotEmpty(dsaTransparency)) {
final String encodedTransparencies = encodeTransparenciesAsString(dsaTransparency);
if (StringUtils.isNotBlank(encodedTransparencies)) {
dsaRequestParams.put("dsatransparency", encodedTransparencies);
}
}

return dsaRequestParams;
}

private static String encodeTransparenciesAsString(List<ExtRegsDsaTransparency> transparencies) {
return transparencies.stream()
.filter(YieldlabBidder::isTransparencyValid)
.map(YieldlabBidder::encodeTransparency)
.collect(Collectors.joining(TRANSPARENCY_TEMPLATE_DELIMITER));
}

private static boolean isTransparencyValid(ExtRegsDsaTransparency transparency) {
return StringUtils.isNotBlank(transparency.getDomain())
&& transparency.getDsaParams() != null
&& CollectionUtils.isNotEmpty(transparency.getDsaParams());
}

private static String encodeTransparency(ExtRegsDsaTransparency transparency) {
return TRANSPARENCY_TEMPLATE.formatted(transparency.getDomain(),
encodeTransparencyParams(transparency.getDsaParams()));
AntoxaAntoxic marked this conversation as resolved.
Show resolved Hide resolved
}

private static String encodeTransparencyParams(List<Integer> dsaParams) {
return dsaParams.stream().map(Objects::toString).collect(Collectors.joining(
TRANSPARENCY_TEMPLATE_PARAMS_DELIMITER));
}
Comment on lines +285 to +306
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some differences with GO logic:

  • transparency0 - invalid, transparency1 - valid
    Java: "transparency1"
    GO: "~~transparency1"
  • transparency0 - valid, transparency1 - invalid, transparency2 - valid
    Java: "transparency0~~transparency2"
    GO: "transparency0~~~~transparency2"

Similar problems with ~ separator:

  • If transparency1.dsaParams is empty
    Java: "transparency0~~transparency2"
    GO: "transparency0~~domain1~~~transparency2"
  • ...

And so on :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me the Java implementation looks correct, and the GO version incorrect.


private static MultiMap resolveHeaders(Site site, Device device, User user) {
final MultiMap headers = MultiMap.caseInsensitiveMultiMap()
.add(HttpUtil.ACCEPT_HEADER, HttpHeaderValues.APPLICATION_JSON);
Expand Down Expand Up @@ -339,7 +413,8 @@ private Bid.BidBuilder addBidParams(YieldlabResponse yieldlabResponse, BidReques
.dealid(resolveNumberParameter(yieldlabResponse.getPid()))
.crid(makeCreativeId(bidRequest, yieldlabResponse, matchedExtImp))
.w(resolveSizeParameter(yieldlabResponse.getAdSize(), true))
.h(resolveSizeParameter(yieldlabResponse.getAdSize(), false));
.h(resolveSizeParameter(yieldlabResponse.getAdSize(), false))
.ext(resolveExtParameter(yieldlabResponse));

return updatedBid;
}
Expand Down Expand Up @@ -408,4 +483,21 @@ private String makeNurl(BidRequest bidRequest, ExtImpYieldlab extImpYieldlab, Yi
yieldlabResponse.getAdSize(),
uriBuilder.toString().replace("?", ""));
}

private ObjectNode resolveExtParameter(YieldlabResponse yieldlabResponse) {
final YieldlabDigitalServicesActResponse dsa = yieldlabResponse.getDsa();
if (dsa == null) {
return null;
}
final ObjectNode ext = mapper.mapper().createObjectNode();
final JsonNode dsaNode;
try {
dsaNode = mapper.mapper().valueToTree(dsa);
} catch (IllegalArgumentException e) {
logger.error("Failed to serialize DSA object for adslot {}", yieldlabResponse.getId(), e);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this error should be returned in the response, not just logged

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would require a big refactoring of the existing implementation as there is no support for returning valid bids along with failed bids. Yieldlab did not write the existing implementation, and we feel it is beyond the scope of this ticket to be refactoring the existing implementation. Our goal is simply to add DSA support since this is a legal requirement. We are fine with not reporting failures.

return null;
}
ext.set("dsa", dsaNode);
return ext;
}
}
26 changes: 26 additions & 0 deletions ...main/java/org/prebid/server/bidder/yieldlab/model/YieldlabDigitalServicesActResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package org.prebid.server.bidder.yieldlab.model;

import lombok.AllArgsConstructor;
import lombok.Value;

import java.util.List;

@AllArgsConstructor(staticName = "of")
@Value(staticConstructor = "of")
public class YieldlabDigitalServicesActResponse {

String behalf;

String paid;

Integer adrender;

List<Transparency> transparency;

@AllArgsConstructor(staticName = "of")
@Value(staticConstructor = "of")
public static class Transparency {
String domain;
List<Integer> dsaparams;
}
}
2 changes: 2 additions & 0 deletions src/main/java/org/prebid/server/bidder/yieldlab/model/YieldlabResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,6 @@ public class YieldlabResponse {
Integer did;

String pvid;

YieldlabDigitalServicesActResponse dsa;
}
Loading
Loading