Daniel Nijkamp ©
Last Updated:
Project Information
Scripts
Project Information
The original idea for the project was just to learn how lists work and how classes can communicate with each other, but since i had alot of time and wanted to do alot i decided to keep upgrading the original project and make it into a console application. Overal the development process was smooth with constant developments in how i approach a problem and how to fix something.
When you open the program you will see the login part of the application, alongside a ASCII image of the SCP logo. you can choose any of the Users, each user has a security clearance level so low level users can only access SCPs with accesslevel of 0 or 1. while the highest one can access every SCP having full access to all of them
The application checks if the password is correct, theres no encryption or anything like that i just wanted to make a log-in screen, the passwords can be found in the Data.json file.
After you have logged in it will check if there is a json file with the database, you just have to type Y. i wanted to add a feature which would make it so you could generate the SCP's and then it would be saved in a JSON file but i never did it so this doesnt do anything, then you can see how fast it generates in real time, it asynchronously scrapes information via HTTPS requests via the Official SCP wiki
Here you can see i limited the generations to 1 series or 999 SCP's 1-999. it will show how long it took and then you can access a SCP, either a see a entire series which doesnt do alot since i didnt expand it or you can choose to search a specific SCP. i would reccomend going with 2. Then once you have typed 2 you can type the number of which SCP you want to access. then it will show the description and the special containment procedure
Overal i was very happy about the end result even though i didnt finish alot of what i wanted to implement as extra features, i learned alot about HTTP requests, JSON serialization, Async & TAP (Task-based asynchronous programming), Tuples, Lists & Dictionaries, String formatting and more about console interaction. When i came back to the project writing and documenting alot for this page i also realized i could optimize it further using multi-threading. Multi-threading + TAP is something i will definitly look into in the near future since i really want to find out how fast it can be then
Scripts
First thing the program script does is display the users, This will be done in the DisplayFetchedUsers function here it will display the info from the GetUsers function. in this function the JSON deserialization happens, the function reads the JSON files and converts it into a User Object. here is a code snippet showing the 2 functions:
When the Users are collected the program exists the GettingCurrentUsers state/loop and goes into the GettingPassword state/loop. here it Uses multiple functions to recieve user input from the console but it primarly uses the Login function where it checks if the password matches what is given as a input. here is a code snippet of that function:
When logged in, the scripts calls the Fetch_SCP_Library which generates the entire archive, but then the program has to make sure the user can interact with the archive. the program.cs script has a function called DisplaySCP which just gets the specified SCP. it first goes in a list which contains lists of SCP objects, there it does a bunch of checks to check for any errors and such things, then just writes it to the console. here is the function:
Both the SCP.cs and User.cs scripts are pretty simple since they are just objects that hold simple info and nothing more, they do both have Newtonsoft.Json. The User script uses it to make json into objects. altough the SCP.cs script has the same JSON properties in it the script doesnt do anything with it, since i never got around to implement the archive being able to save all the generated SCPs
This is were the magic happens, this script used to be slow, not work or have bugs that would crash the program. generating 8000 thousand used to take hours? its a guess since i left it running once and it was no where near done and going for 20 minutes. now because of multiple optimalization it generates everything around ~5-7 minutes! the script first was using synchronous functions and was too slow and now used async functions to do multiple tasks at the same time.
This is the GenerateSCP function which is async and has the return type Task<SCP>. This allows it to be asynchronously meaning we can run multiple of these functions at the same time. The first thing the script does initialize a bunch of HTTP stuff and format the URL we need to extract data from. then it retrieves the HTML document asynchronously via HTMLagilitypack. then it goes to a sub-section of website with the id: page-content .Here lies all the info we need. then what the script does is loop through each html element in the page-content div and checks if it contains what we want. this is further optimized using a switch case. Then the final step of the function: Make a new SCP object and insert everything we got from the website. Typing it makes it sound very complex but if you see the code it is very logical and understandable
Now this function is the one responsible for calling the GenerateSCP function and formatting the entire archive. first thing it does is have a loop from 1 to 8, representing the series. so it goes from 0-999 then from 1000 to 1999 and so on and so on. Then it makes a dictionary with the values: <int, Task<SCP>>. This tasklist will recieve the completed tasks that return SCP objects filled with info. Then we have the actual loop that goes from 0-999 and calls our GenerateSCP function. but if you look at the code snippet you will see something weird. Why is there a Thread.Sleep function? doing this will slow down the entire thread and make all the tasks slower! this is done because when i ran the script without it. the https requests were made so fast that the server declined the request thinking it was being DDOS'ed!. Anyway then the function waits until all tasks are complete and then sorts the task lists by number, because it was generated asynchronously its completly random. then after its sorted the SCP's in the dictionary are added to a list and that list is added to the SCP_Series list. there everything is stored and accessed
