Websurfx - An open source alternative to Searx which aggregates results from other search engines (metasearch engine) without ads while keeping privacy and security in mind.
from neon_arch@programming.dev to rust@programming.dev on 07 Sep 2023 10:04
https://programming.dev/post/2678496

Introduction

Hello everybody, About 5 months ago I started building an alternative to the Searx metasearch engine called Websurfx which brings many improvements and features which lacks in Searx like speed, security, high levels of customization and lots more. Although as of now it lacks many features which will be added soon in futures release cycles but right now we have got everything stabilized and are nearing to our first release v1.0.0. So I would like to have some feedbacks on my project because they are really valuable part for this project.

In the next part I share the reason this project exists and what we have done so far, share the goal of the project and what we are planning to do in the future.

Why does it exist?

The primary purpose of the Websurfx project is to create a fast, secure, and privacy-focused metasearch engine. While there are numerous metasearch engines available, not all of them guarantee the security of their search engine, which is critical for maintaining privacy. Memory flaws, for example, can expose private or sensitive information, which is never a good thing. Also, there is the added problem of Spam, ads, and unorganic results which most engines don’t have the full-proof answer to it till now. Moreover, Rust is used to write Websurfx, which ensures memory safety and removes such issues. Many metasearch engines also lack important features like advanced picture search, which is required by many graphic designers, content providers, and others. Websurfx attempts to improve the user experience by providing these and other features, such as providing custom filtering ability and Micro-apps or Quick results (like providing a calculator, currency exchanges, etc. in the search results).

Preview

Home Page

Search Page

404 Page

What Do We Provide Right Now?

In Future Releases

We are planning to move to leptos framework, which will help us provide more privacy by providing feature based compilation which allows the user to choose between different privacy levels. Which will look something like this:

Goals

Benchmarks

Well, I will not compare my benchmark to other metasearch engines and Searx, but here is the benchmark for speed.

Number of workers/users: 16
Number of searches per worker/user: 1
Total time: 75.37s
Average time per search: 4.71s
Minimum time: 2.95s
Maximum time: 9.28s

Note: This benchmark was performed on a 1 Mbps internet connection speed.

Installation

To get started, clone the repository, edit the config file, which is located in the websurfx directory, and install the Redis server by following the instructions located here. Then run the websurfx server and Redis server using the following commands.

git clone https://github.com/neon-mmd/websurfx.git
cd websurfx
cargo build -r
redis-server --port 8082 &
./target/debug/websurfx

Once you have started the server, open your preferred web browser and navigate to http://127.0.0.1:8080 to start using Websurfx.

Check out the docs for docker deployment and more installation instructions.

Call to Action: If you like the project then I would suggest leaving a star on the project as this helps us reach more people in the process.

<img alt="" src="https://programming.dev/pictrs/image/fa94143a-8d39-4a71-be35-ded906a7920a.webm"> “Show your love by starring the project”

Project Link:

https://github.com/neon-mmd/websurfx

#rust

threaded - newest

Eikichi@lemmy.ml on 07 Sep 2023 10:34 next collapse

Ty ! Once on my PC, I will definitely give it a try,

neon_arch@programming.dev on 07 Sep 2023 21:21 collapse

Thanks for trying out our project :). If you need any help, please feel free to open an issue at our project.

github.com/neon-mmd/websurfx/issues

asdfasdfasdf@lemmy.world on 07 Sep 2023 11:24 next collapse

Based on the benchmarks, it looks like it’s not running searches concurrently?

neon_arch@programming.dev on 07 Sep 2023 21:19 collapse

Thanks for pointing this out, I just improved this by upgrading the algorithm to use tokio::spawn so I think I will update this benchmarks soon.

asdfasdfasdf@lemmy.world on 07 Sep 2023 23:31 collapse

Nice! Happy to help.

neon_arch@programming.dev on 08 Sep 2023 15:44 collapse

Thanks, I am very grateful for that :).

smollittlefrog@lemdro.id on 07 Sep 2023 12:29 next collapse

Hey, which engines are intended to be supported in the future?

Under src/engines I could find files for duckduckgo and searx. Are both already fully supported? Do you intend to support Google and Yandex in the future?

neon_arch@programming.dev on 07 Sep 2023 21:24 collapse

Hello :).

The searx and duckduckgo engines are fully supported right now, and we are already looking forward to having more engines supported as well. Just, that we are in need of some help with the process because you know there are too many engines too work on :).

orizuru@lemmy.sdf.org on 07 Sep 2023 22:29 next collapse

Interesting, I’ll be keeping an eye on this. Thanks for sharing!

I’m currently self hosting SearXNG. The must-have features for me are the custom filters and the actively maintained docker image. Will definitely give it a go if they get implemented.

neon_arch@programming.dev on 08 Sep 2023 15:51 collapse

Thanks for taking a look at my project :).

The custom filter is about to be added soon, just the PR for it waiting to be merged. Once that is merged. We will have custom filer feature available. Though about the docker image feature it is available, I would suggest taking a look at this section of the docs:

github.com/neon-mmd/websurfx/…/installation.md#do…

Here we cover on how to get our project deployed via docker.

orizuru@lemmy.sdf.org on 09 Sep 2023 08:37 collapse

Ah cool, thanks!

Will definitely try it now. It’s good to have options (Searx just recently became unmaintained).

Are there any plans to have an official docker hub image? I’m asking because my workflow involves keeping the containers up to date with watchtower.

neon_arch@programming.dev on 12 Sep 2023 17:04 collapse

Sorry for the delay in the reply.

Ok, thanks for suggesting this out. I have not thought about particularly in this area, but I would be really interested to have the docker image uploaded to docker hub. The only issue is that the app requires that the config file and blocklist and allowlists should be included within the docker hub. So the issue is that if a prebuilt image is provided, then is it possible to edit it within the docker container ?? If so then it is ok, otherwise it would still be good, but it would limit the usage to users who are by default satisfied by the default config. While others would still need to build the image manually, which is not very great.

Also, As side comment in case you have missed this. Some updates on the project:

  • We have just recently got the custom filter lists feature merged. If you wish to take a look at this PR, here.
  • Also, recently there has been ongoing on getting new themes added, and an active discussion is going on that topic and some themes’ proposal have been placed. Here is a quick preview of one of the theme and what it might look like:

Home Page

<img alt="" src="https://programming.dev/pictrs/image/1027d7da-bd51-4603-a77e-3aa723970010.png">

Search Page

<img alt="" src="https://programming.dev/pictrs/image/1f7a69ac-df02-446f-964f-4abbefcb3ada.png">

orizuru@lemmy.sdf.org on 12 Sep 2023 18:09 collapse

Sorry for the delay in the reply.

No need to apologize! Thank you for working on this. :)

The only issue is that the app requires that the config file and blocklist and allowlists should be included within the docker hub. So the issue is that if a prebuilt image is provided, then is it possible to edit it within the docker container ?? If so then it is ok, otherwise it would still be good, but it would limit the usage to users who are by default satisfied by the default config. While others would still need to build the image manually, which is not very great.

I’m not familiar with the websurfix codebase, but I don’t see why it wouldn’t work.

I’m currently self-hosting SearXNG on a VPS, but I started by having it just locally. The important bit of that blog post is this:

docker run -d --rm \
              -d -p 8080:8080 \
              -v "${HOME}/searxng:/etc/searxng" \
              -e "BASE_URL=http://localhost:8080/" \
              searxng/searxng

I use the -v flag to mount a directory in my home to the config directory inside the docker container. SearXNG then writes the default config files there, and I can just edit them normally on ~/searxng/.

By using a mounted volume like this, the configs are persistent, so I can restart the docker container without losing them.

neon_arch@programming.dev on 13 Sep 2023 16:18 collapse

Ahh, I see, Why didn’t I remember this before that I can do something like this. Thanks for the help :). Actually the thing is I am not very good at docker, and I am in the process of finding someone who can actually work on in this area like for example reducing build times, caching, etc. One of the things we want to improve right now is reducing build time like I am using layered caching approach but still it takes about 800 seconds which is not very great. So if you are interested then I would suggest making a PR at our repository. We would be glad to have you as part of the project contributors. And Maybe in future as the maintainer too. Currently, the Dockerfile looks like this:

FROM rust:latest AS chef
# We only pay the installation cost once,
# it will be cached from the second build onwards
RUN cargo install cargo-chef

WORKDIR /app

FROM chef AS planner
COPY . . 
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json

# Build application
COPY . .
RUN cargo install --path .

# We do not need the Rust toolchain to run the binary!
FROM gcr.io/distroless/cc-debian12
COPY --from=builder /app/public/ /opt/websurfx/public/
COPY --from=builder /app/websurfx/config.lua /etc/xdg/websurfx/config.lua # -- 1
COPY --from=builder /app/websurfx/config.lua /etc/xdg/websurfx/allowlist.txt # -- 2
COPY --from=builder /app/websurfx/config.lua /etc/xdg/websurfx/blocklist.txt # -- 3
COPY --from=builder /usr/local/cargo/bin/* /usr/local/bin/
CMD ["websurfx"]

Note: The 1,2 and 3 marked in the Dockerfile are the files which are the user editable files like config file and custom filter lists.

orizuru@lemmy.sdf.org on 13 Sep 2023 16:52 collapse

You’re welcome!

I can have a look in my free time for fun. Will let you know if I manage to do it. 😅

neon_arch@programming.dev on 13 Sep 2023 17:48 collapse

Ok no problem :). If you need any help regarding anything, just DM us/me here or at our Discord server. We would be glad to help :).

lynx@sh.itjust.works on 08 Sep 2023 21:52 next collapse

How do you rank the results from different search engines?

neon_arch@programming.dev on 10 Sep 2023 16:38 collapse

Hello again :)

Sorry for the delayed reply.

Right now, we do not have ranking in place, but we are planning to have it soon. Our goal is to make it as organic as possible, so you don’t get unrelated results when you query something through our engine.

What the project does is it takes the user query and various search parameters if necessary and then passes it to the upstream search engines. It then gets its results with the help of a get request to the upstream engine. Once all the results are gathered, we bring it to a form where we can aggregate the results together and then remove duplicate results from the aggregated results. If two results are from the same engine, then we put both engine’s name against the search result. That’s what is all going, in simple terms :slight_smile: . If you have more doubts. Feel free to open an issue at our project, I would be glad to answer.

hascat@programming.dev on 09 Sep 2023 00:28 next collapse

Ad-Free Results

How are you compensating the search engines you query?

neon_arch@programming.dev on 10 Sep 2023 16:43 collapse

Hello again :)

Sorry for the delayed reply.

It is essentially, how we are achieving the Ad-free results is when we fetch the results from the upstream search engines. We then take the ad results from all of them, bring it to a form where it is aggregatable and then aggregate it. That’s how we achieve it.

caralice@mas.to on 09 Sep 2023 13:11 collapse

@neon_arch is searx not open source??

lynx@sh.itjust.works on 09 Sep 2023 19:50 next collapse

It is github.com/searx/searx but you should use searxng instead: github.com/searxng/searxng

neon_arch@programming.dev on 10 Sep 2023 16:31 collapse

Yes, it is, but I just wanted to emphasize that my project is also open source because if I don’t add this then it can raise some doubts whether it is open source or not. So to make it clear, I added it.