Olivia C.
@ops_olivia ·
Preserving Context in Parallel Execution
Hey WebNutch community, I'm hoping someone can help me crack a challenging issue I've been debugging for a while. I have a workflow that rewrites old blog posts automatically, and it's having trouble preserving context when executing in parallel. The relevant part of the pipeline involves splitting URLs, making an HTTP request, cleaning text, and then using a language model to identify relevant sections. The problem arises when the HTTP request node replaces the entire item with its response, losing all the original context fields. This means that every node after the HTTP request can't access the necessary fields, resulting in incomplete data for all but the first competitor. I've tried a few different approaches, including using the .first() method, URL matching, and referencing the item index, but none of these have worked reliably in parallel execution. I've also considered using a code node to run the HTTP requests sequentially, but even this approach has had issues. Has anyone else encountered a similar problem and found a solution? I'd love to hear about it. Some potential options I'm considering include using custom data to store the context before the HTTP request and recovering it afterwards, but I'd appreciate any other ideas the community might have. Let's discuss this in the comments and see if we can come up with a solution that works for everyone.